Skip to content

Jmni/zarr validation - #607

Open
jaynye wants to merge 9 commits into
jmni/zarr-registryfrom
jmni/zarr-validation
Open

Jmni/zarr validation#607
jaynye wants to merge 9 commits into
jmni/zarr-registryfrom
jmni/zarr-validation

Conversation

@jaynye

@jaynye jaynye commented Sep 3, 2026

Copy link
Copy Markdown
  • Per-episode calibration with per-embodiment fall-back
  • Optional camera metadata, such as model and distortion
  • Added structural validation to prevent the loading of example or test data from vendors

A validation script with the heuristic checks:

  1. embodiment, integer total frames, fps, task_name, features
  2. timestamps must be monotonic
  3. basic rule against synthetic calibration, intrinsics, or poses

The following are checks configurable to be optional:

  1. camera_coverage, for data that does not have calibrated wrist cameras
  2. annotation_coverage, for data that is not densely annotated
  3. schema_version, data_status, calibration, for old data

jaynye and others added 9 commits September 2, 2026 20:45
Calibration is a measurement of the rig that recorded one episode, so it has to
travel with the episode instead of living in a class constant. Add the
`calibration` attribute block and the single reader every consumer goes
through.

The block names one reference frame and expresses every pose in it:
`cameras[c].ref_T_cam` is a camera pose, `arm_bases[side]` is `ref_T_armbase`.
`Calibration.base_T_cam(side)` composes the two into what the EVA transform
pipeline consumes.

No stored episode needs a rewrite. `read_calibration` prefers the block and
falls back to lifting the legacy `intrinsics`/`extrinsics` pair, whose
per-arm values are the front camera's pose in each arm base, so the lifted
reference frame is `camera:front_1` and `arm_bases` is their inverse.
`ZarrWriter` writes the block and derives the legacy pair from it, so readers
that predate the block keep working.

`ZarrEpisode`, `ZarrDataset` and the inspector now read through the shim
rather than parsing the attributes themselves, which also drops two duplicate
3x3-to-3x4 normalizations.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rCh1QmnyqgPZ5TJpsfMQb
`intrinsics` only had to be non-empty, so `eva_to_zarr.py` calibrates one
camera while writing three image streams and passes. Coverage closes that: the
declared `images.<camera>` keys are checked against the cameras that carry a K.

The rule is opt-in. Every EVA episode in the corpus fails it today, so
`strict=False` warns and `strict=True` raises. Flipping the default before the
corpus is fixed would turn a good check into an outage.

`uncalibrated_cameras` lives beside the calibration reader so the validator can
reuse the same rule rather than restate it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rCh1QmnyqgPZ5TJpsfMQb
The transform pipeline expressed actions in the camera frame using
`Eva.EXTRINSICS`, a class constant, and ignored what the episode stored. Two
vendors on one platform have two rigs, and a class constant cannot tell them
apart, so the factorization is unsound until the training path reads the
episode.

`ZarrDataset` now puts each arm's `base_T_cam` from the episode calibration
into every sample, and a transform's `extra_batch_key` becomes a fallback that
fills only keys the sample does not already carry. `Eva.EXTRINSICS` still
covers an episode that declares no extrinsics.

This changes numbers only for an episode whose stored rig differs from the
constant. A test asserts the two agree: an episode holding `Eva.EXTRINSICS`
and an episode holding none produce the same actions, which is every EVA
episode in the corpus today. Two further tests show a second rig moves the
actions and lands exactly where that rig predicts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rCh1QmnyqgPZ5TJpsfMQb
Collecting calibration is cheap and time-critical; consuming it is expensive
and deferrable. A vendor who ships 10k episodes and later turns out to need
distortion coefficients cannot supply them, because the rig has moved.

Each camera in the block now declares `model` and `distortion`, defaulting to
PINHOLE with no coefficients. The coefficient count is checked against the
model, so a KANNALA_BRANDT camera with five coefficients or a pinhole camera
with any is an error at ingest.

No projection site reads either field. Nothing under `algo/` or `models/`
reads intrinsics at all, so this changes no trained weight.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rCh1QmnyqgPZ5TJpsfMQb
`python -m egomimic.rldb.zarr.validate <path> [--strict]` checks one episode
against `schema/episode_v3.yaml`. The rules live in the schema file; the module
reads them and holds none of its own, so a contributor can read and diff the
contract instead of reverse-engineering it from Python.

It replaces the 300-line `validate_episode()` pasted into CONTRIBUTING_DATA.md,
which is the only validator a contributor can run today and which passed an
all-identity episode with 28 checks OK and 0 errors.

Array rules resolve their widths through the registry, so `arm_dof` and the
keypoint topology come from `platforms.yaml` and `end_effectors.yaml` rather
than being restated. `{side}` expands over the arms the arity declares, and a
rule applies only when its `when` conditions hold, so a human episode is never
asked for a gripper.

Rules the corpus does not meet yet are declared `required: strict`: a warning
by default and an error under `--strict`. Flipping them first would turn a good
check into an outage, since the intrinsics coverage rule alone fails every
`eva_fold` episode.

`total_frames` stays the sole authoritative length: axis 0 is a lower bound so
a padded tail passes, and every other axis is exact.

Adds `ResolvedEmbodiment.arity` and `.sides`, which the array rules need to
know whether an episode owes one arm or two.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rCh1QmnyqgPZ5TJpsfMQb
Six rules that turn the failure modes we have actually been shipped into
findings at ingest instead of a policy that trains badly six weeks later.

- pose_degeneracy: a track that never moves, or that holds an exact identity
  rotation on more than 1% of frames, is a placeholder, not a measurement.
- calibration_degeneracy: an exactly-identity extrinsic puts the camera at the
  arm base. It survives every other check, because the overlay it draws still
  looks plausible, bunched at the image centre.
- intrinsics_signature: `fx == fy == W` with the principal point at the exact
  image centre is a synthesized camera. The `fx == W` conjunct is what keeps
  our own rectified Aria K, fx 266.5 at W 640, out of the net.
- timestamps: strictly increasing, and a warning when the step is a power of
  two of at least 64 ns, which is float64 seconds converted to nanoseconds. A
  second stored time base is an error.
- annotation_coverage: annotations cover 90% of the episode or the tail is
  trimmed. Opt-in, since EVA episodes pass no annotations at all.
- annotation_text: no delimiter-encoded metadata. A skill taxonomy defined
  from one vendor's data is a taxonomy we will regret.

Every rule and its threshold lives in the schema file.

Run against the Sharpa sample the rules reproduce each measurement in the
review by hand: 370 identity frames on all three pose tracks, identity
extrinsics, the synthesized 480-pixel K on all three cameras, three duplicate
timestamps under 256 ns quantization, 76% annotation coverage, and the
` | Skill: pick` suffix. The validator that shipped in CONTRIBUTING_DATA.md
passes that episode with 28 checks OK and 0 errors.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rCh1QmnyqgPZ5TJpsfMQb
A sample sent to show the shape of a delivery is real data in the schema sense
and not real data in the training sense, and until now nothing downstream could
tell the two apart. Sharpa reached for this with six free-text `*_status`
attributes that nothing reads.

`data_status` is `complete` or `structural_sample`. The writer records it and
rejects anything else. A non-complete episode gets no staging row and is
skipped by the resolver, so it cannot reach a training run by any path.

An episode written before the attribute existed reads as `complete`: the corpus
predates the distinction and every episode in it was delivered as finished
data. The validator therefore asks for the attribute only under `--strict`,
while an unknown value is always an error.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rCh1QmnyqgPZ5TJpsfMQb
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown

Claude Code Review

Review of PR #607: Jmni/zarr validation

Summary

This PR introduces an embodiment registry system (YAML-driven platforms/end-effectors/aliases), a per-episode calibration block with a legacy shim, coordinate-convention docs (A_T_B), and structural zarr validation. It also adds CI. The changes are broad but mostly well-scoped and internally consistent.

Key concerns

1. Scope mismatch with PR description

The description says "per-episode calibration with fallback" and "structural validation," but the diff also renames extrinsics semantics repo-wide (extra_batch_key fallback semantics changed from .update.setdefault), refactors the entire embodiment system into a registry, and renames T_RGB_CPFRGB_T_CPF. Each of these is individually reasonable but should ideally be split. As-is, this is high-risk to land in one shot.

2. Silent semantic change in extra_batch_key fallback

ActionChunkCoordinateFrameTransform and CartesianWithGripperCoordinateTransform previously called batch.update(extra_batch_key) (extras win) and now call _apply_fallbacks (batch wins via setdefault). This is the correct new semantics for episode calibration, but:

  • It reverses precedence for any existing sample that happens to carry a key like left_extrinsics_pose (unlikely, but the pipelines also renamed keys to left_base_T_cam_pose, so any lingering upstream code producing the old name will now be silently ignored instead of overridden).
  • No test asserts the new precedence (per-episode wins over class default).

Please add an explicit test: given Eva.EXTRINSICS as fallback and a per-sample left_base_T_cam_pose, the sample value must be used.

3. Legacy shim correctness

The doc claims: "predates the block reaches the same value through the shim... its reference frame is camera:front_1, so arm_bases[side] is the inverse of the stored extrinsics[side]."

But Eva.EXTRINSICS is documented as base_T_cam (camera pose in arm-base frame). If the reference frame is camera:front_1, then arm_bases[side] = ref_T_armbase = cam_T_base = inv(base_T_cam). ✓ math checks out.

However, ActionChunkCoordinateFrameTransform with inverse=True computes target_T_chunk = inv(ref_T_target) @ ref_T_chunk. The new key left_base_T_cam_pose fed as target_world means the transform is computing inv(base_T_cam) @ (base_T_chunk) = cam_T_chunk. That matches previous behavior since Eva.EXTRINSICS["left"] is unchanged. ✓

But — please verify with a numerical regression test on one real EVA episode that a full transform pipeline produces bit-identical outputs before/after this rename. The variable renames touch enough of the projection path that a golden-value test is warranted.

4. Alias-write policy needs enforcement earlier

test_aliases_are_a_read_shim_only verifies ZarrWriter rejects aria_bimanual. Good. But get_embodiment_id("aria_bimanual") silently maps to HUMAN_BIMANUAL. This means anyone constructing metadata without going through ZarrWriter (SQL insert paths, tests, notebooks) can write a mislabeled row. Recommend also validating embodiment at SQL-insert time against EMBODIMENT.__members__ (not the alias table).

5. Registry file location under rldb/embodiment/

Putting YAML config files inside a Python package works via Path(__file__).parent but is fragile for wheel builds / editable installs on some setups. Consider importlib.resources or moving them to egomimic/hydra_configs/ alongside other configs.

6. T_RGB_CPFRGB_T_CPF rename

This is a name-only change (the matrix values are unchanged), and the doc convention says A_T_B maps B → A. The old name T_RGB_CPF was ambiguous; RGB_T_CPF (maps CPF → RGB) is clearer. But grep the codebase for any external notebooks, saved configs, or checkpoints that reference T_RGB_CPF by attribute — Aria gaze viz is an active research path.

Suggestions

  • Split the PR (or at minimum, land the coordinate-convention doc + rename first, then registry, then calibration/validation). Landing all three plus CI in one PR makes bisecting a training regression painful.
  • Add a golden-value regression test for the EVA bimanual pipeline: load one committed toy episode → run get_transform_list("cartesian") → assert numeric equality with a checked-in golden output.
  • Add a norm-stats regression check: since the transform key names changed (left_extrinsics_poseleft_base_T_cam_pose), any cached norm-stats file keyed by these names will silently miss. Confirm norm-stats are recomputed from scratch or that the affected keys are internal (deleted by KeysToDelete before norm-stat collection).
  • Test the fallback precedence directly (see Concern Main fixes move to roger's pipeline branch #2).
  • CI: ruff check egomimic is narrower than the repo — add scripts/, tests/ if they exist at top level. Also consider a ruff format --check.
  • The truncated diff means I can't review validate_zarr.py itself, which is nominally the point of the PR — please flag when the actual validator is reviewable.

Verdict: Request Changes

The direction is good and the registry/calibration design is clean, but the PR bundles too many concerns for a single review, lacks a numerical regression test for the extrinsics rename, and the silent fallback-precedence flip needs an explicit test. I'd like to see either a split or the


Reviewed by Claude · Review workflow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant